Visual search assistance for an occupant of a vehicle

ABSTRACT

A visual search assistance system and related operating method are disclosed. An exemplary embodiment of the method dynamically determines a desired line-of-sight vector from an occupant of a vehicle to a visual target, and dynamically determines, with an onboard tracking system of the vehicle, an actual line-of-sight vector for the occupant of the vehicle. The actual line-of-sight vector is temporally associated with the desired line-of-sight vector. The method compares the actual line-of-sight vector against the desired line-of-sight vector to obtain a difference measurement, and generates a notification when the obtained difference measurement exceeds a threshold value. The notification provides guidance to redirect a gaze direction of the occupant toward the visual target.

TECHNICAL FIELD

Embodiments of the subject matter described herein relate generally tovehicle systems and subsystems. More particularly, embodiments of thesubject matter relate to an aircraft crew alerting and visual guidancesystem.

BACKGROUND

Instrumentation, display, and alerting systems for vehicles aregenerally known. Such systems are particularly important for safeoperation of vehicles such as aircraft, spacecraft, and watercraft. Inan aircraft context, for example, pilots must maintain situationalawareness of the environment both inside and outside the cockpit.External situational awareness is important for purposes of identifyingneighboring aircraft traffic, visually identifying landmarks, locatingairports, determining weather patterns, and the like. Situationalawareness inside the cockpit is also important for purposes ofidentifying and reading displays, instrumentation, alerts, and the like.

Visually locating neighboring aircraft in flight can be challenging dueto the cluttered ground background, sunlight or other lightingconditions, weather, and the large three-dimensional viewspace that istypically involved. Challenges for locating ground-based landmarks suchas airports are similar, especially when the pilot is operating in newor unfamiliar areas. Likewise, it may be difficult for a pilot toquickly and easily identify and view a display element, alert message,indicator light, or other element inside the cockpit that requiresattention. Visually identifying a specific item within an aircraftcockpit can be particularly challenging when the cockpit contains alarge amount of instrumentation, controls, switches, displays, andindicator lights.

Accordingly, it is desirable to have an effective, accurate, andpractical solution to assist an operator or occupant of a vehicle (suchas an aircraft) in visually finding and identifying objects that may beinside the vehicle or outside the vehicle. In addition, it is desirableto have a system onboard a vehicle that alerts an occupant of thevehicle when necessary to direct the attention of the occupant to avisual target. Furthermore, other desirable features and characteristicswill become apparent from the subsequent detailed description and theappended claims, taken in conjunction with the accompanying drawings andthe foregoing technical field and background.

BRIEF SUMMARY

A visual search assistance method for an occupant of a vehicle isdisclosed herein. An exemplary embodiment of the method dynamicallydetermines a desired line-of-sight vector from an occupant of thevehicle to a visual target, and dynamically determines, with an onboardtracking system of the vehicle, an actual line-of-sight vector for theoccupant of the vehicle. The actual line-of-sight vector is temporallyassociated with the determined desired line-of-sight vector. The methodcontinues by comparing the determined actual line-of-sight vectoragainst the determined desired line-of-sight vector to obtain adifference measurement. A notification is generated when the obtaineddifference measurement exceeds a threshold value. The notificationprovides guidance to redirect a gaze direction of the occupant towardthe visual target.

Also disclosed is a tangible and non-transitory computer readable mediumhaving computer executable instructions stored thereon and capable ofperforming a method when executed by a processor. An exemplaryembodiment of the method dynamically obtains a desired line-of-sightvector from an occupant of an aircraft to a visual target, anddynamically obtains, from an onboard tracking system of the aircraft, anactual line-of-sight vector for the occupant of the aircraft. The actualline-of-sight vector is temporally associated with the determineddesired line-of-sight vector. The actual line-of-sight vector iscompared against the desired line-of-sight vector to obtain a differencemeasurement. The method continues by generating a notification when theobtained difference measurement exceeds a threshold value, thenotification providing guidance to redirect a gaze direction of theoccupant toward the visual target.

A computing system for a vehicle is also presented herein. An exemplaryembodiment of the computing system includes a processor architecturehaving at least one processor device, and memory havingcomputer-executable instructions stored thereon. When the instructionsare executed by the processor architecture, the computing systemoperates an onboard tracking system of the vehicle to dynamically obtaina current line-of-sight vector for an occupant of the vehicle. Thecomputing system compares the current line-of-sight vector against acurrent desired line-of-sight vector from the occupant of the vehicle toa visual target, wherein the current line-of-sight vector is temporallyassociated with the current desired line-of-sight vector. The computingsystem provides information when the difference measurement exceeds athreshold value. The information is intended for the occupant of thevehicle, and the information provides guidance to redirect a gazedirection of the occupant toward the visual target.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the subject matter may be derived byreferring to the detailed description and claims when considered inconjunction with the following figures, wherein like reference numbersrefer to similar elements throughout the figures.

FIG. 1 is a simplified schematic representation of an exemplaryembodiment of an onboard system that provides visual search assistanceto an occupant or operator of a vehicle;

FIG. 2 is a diagram that depicts an in-flight scenario where the pilotof an aircraft needs assistance to direct his gaze toward a visualtarget of interest;

FIG. 3 is a simplified block diagram representation of an exemplaryembodiment of an onboard aircraft system that provides visual searchassistance to an occupant or operator of the aircraft; and

FIG. 4 is a flow chart that illustrates an exemplary embodiment of avisual search assistance process.

DETAILED DESCRIPTION

The following detailed description is merely illustrative in nature andis not intended to limit the embodiments of the subject matter or theapplication and uses of such embodiments. As used herein, the word“exemplary” means “serving as an example, instance, or illustration.”Any implementation described herein as exemplary is not necessarily tobe construed as preferred or advantageous over other implementations.Furthermore, there is no intention to be bound by any expressed orimplied theory presented in the preceding technical field, background,brief summary or the following detailed description.

Techniques and technologies may be described herein in terms offunctional and/or logical block components, and with reference tosymbolic representations of operations, processing tasks, and functionsthat may be performed by various computing components or devices. Suchoperations, tasks, and functions are sometimes referred to as beingcomputer-executed, computerized, processor-executed,software-implemented, or computer-implemented. It should be appreciatedthat the various block components shown in the figures may be realizedby any number of hardware, software, and/or firmware componentsconfigured to perform the specified functions. For example, anembodiment of a system or a component may employ various integratedcircuit components, e.g., memory elements, digital signal processingelements, logic elements, look-up tables, or the like, which may carryout a variety of functions under the control of one or moremicroprocessors or other control devices.

When implemented in software or firmware, various elements of thesystems described herein are essentially the code segments or executableinstructions that, when executed by one or more processor devices, causethe host computing system to perform the various tasks. In certainembodiments, the program or code segments are stored in a tangibleprocessor-readable medium, which may include any medium that can storeor transfer information. Examples of suitable forms of non-transitoryand processor-readable media include an electronic circuit, asemiconductor memory device, a ROM, a flash memory, an erasable ROM(EROM), a floppy diskette, a CD-ROM, an optical disk, a hard disk, orthe like.

The techniques and technologies described here can be deployed with anyvehicle, including, without limitation: aircraft; watercraft; roadvehicles such as cars, buses, trucks, and motorcycles; spacecraft;trains; subways; specialty equipment (e.g., construction equipment,factory equipment, etc.); trams; and the like. The particularembodiments described below relate to aircraft applications, however,the subject matter is not limited or restricted to such aircraftapplications.

As aircraft become more complex, it is important to keep the flight crewattentive and informed in a timely manner regarding air traffic, alerts,and the like. For example, it is beneficial for a pilot to know thepositioning of other aircraft during flight, and to know the layout andlocation of taxiways and runways during landing and takeoff maneuvers.The embodiments presented here leverage an onboard tracking system thatuses onboard sensors and related processing techniques to determine thegaze direction of an occupant of the aircraft, typically the pilot orco-pilot. As used here, “occupant” is intended to include any persononboard the vehicle. Thus, an occupant may be, without limitation: apilot; a co-pilot; a driver; an operator; a crew member; a passenger; atechnician; a user; an observer; etc.

In accordance with certain embodiments, the onboard tracking systemincludes, cooperates with, or is realized using a near-to-eye (NTE)display system. NTE display has been touted as an effective andinexpensive means of presenting navigation and safety information, andas a replacement for traditional head-up display technologies. NTEdisplay systems use sensors (e.g., infrared light emitting diodes,cameras, or the like) to sense the user's head position and orientationfor purposes of presenting appropriate information on the NTE displayelement. Outside their use in the NTE system, these sensors can be usedto collect detailed information about the pilot's state and actions. Incertain situations, a crew member may not be fully attentive toimportant or critical events in the surrounding environment, such as thepresence of neighboring aircraft in the vicinity of the host aircraft.In such situations, the direction of the person's eye gaze may bedifferent than the desired direction toward a visual target of interest.

Moreover, during flight operations, crew members onboard an aircraftreceive information such as traffic advisories, terrain warnings, systemstatus information, and alerts. An occupant of the aircraft may beexpected to quickly view and comprehend information provided duringflight. The system and methodology described here keeps track ofreceived information that might require visual attention, and calculatesor obtains the expected or desired line-of-sight direction from the crewmember to the visual target. This real-time status information iscorrelated with the crew member's actual view state, as obtained fromthe NTE system, to determine any variation between the desired andactual direction of visual attention. In addition, when the occupant'sactual eye gaze direction deviates from the desired direction toward thevisual target, the onboard system generates an alert (e.g., a visualindicator) that identifies the desired direction of visual attention.

For example, Traffic Advisories (TAs) are issued to facilitate visualacquisition of air traffic in the vicinity of the host aircraft. Pilotsare expected to visually acquire the air traffic and prepare for aResolution Advisory (RA). The system described here can compare thepilot's actual viewing direction against a viewing direction associatedwith an aircraft identified by a TA and generate an alert when it isdetermined that the pilot is not looking in the appropriate direction.If necessary, a visual indicator pointing to the desired visual target(such as directional arrow, a chevron, or the like) can be renderedusing the NTE system and/or other display systems onboard the aircraft.

In accordance with certain embodiments, the visual search assistancesystem employs an eye gaze tracking system and known positioninformation for both the observer and the visual target object. In thisregard, the visual assistance system may leverage global positioningsystem (GPS) data that refers to one point in three-dimensional spacefor each object. The GPS data can be processed to provide a directionalvector from the observer to the visual target. The eye gaze trackingsystem provides a directional vector corresponding to the current gazedirection of the observer, and the system compares the two directionalvectors to provide feedback to the observer to adjust his or her currentgaze direction as needed to find the desired visual target. In practice,visual targets may be, without limitation, gauges, panels, circuitbreakers, or switches that may be visually obstructed. The pilot canthen query the search assistance system through voice activation andreceive audible feedback to correct the eye gaze direction.

Conversely, the pilot may query the system for assistance in identifyingan object seen by the pilot. For example, the pilot may see anunfamiliar airport on the ground, and may want to know the name of theairport. By comparing the directional vector of the pilot's eye gaze tothe nearest known airport intersecting the ground plane with the gazevector, the system can determine the name of the airport and communicatethe name to the pilot.

This particular embodiment may include an eye gaze tracking subsystemmounted in the flight deck and/or a NTE subsystem such as glasses orgoggles worn by the pilot. The eye tracking subsystem (aftercalibration) provides a directional vector for the occupant's gazerelative to the aircraft. The visual assistance system may also leveragea GPS receiver that provides information related to the pilot's position(latitude, longitude, and elevation), as well as time and headinginformation that can be used to estimate the directional vector from theaircraft to the searched visual target. This directional vector canprovide an angle difference, relative to the visual target, inreal-time. The assistance system may also include a speech synthesizerthat can enunciate the magnitude and direction of the angle differenceto the pilot such that the gaze can be corrected to reduce the gazevector error and such that the pilot can quickly and intuitively locatethe visual target of interest.

Referring now to the drawings, FIG. 1 is a simplified schematicrepresentation of an exemplary embodiment of an onboard system 100 thatprovides visual search assistance to an occupant 102 of a vehicle suchas an aircraft. FIG. 1 depicts various onboard aircraft subsystems thatcooperate to provide visual assistance and guidance to the occupant 102(e.g., the pilot, the co-pilot, a crew member, or any observer onboardthe aircraft). This embodiment of the system 100 generally includes,without limitation: an eye and/or head tracking subsystem 104; a visualtarget monitor and/or tracking subsystem 106; a line-of-sight vectorcalculation and analysis module 108; a display subsystem 110; an audiosubsystem 112; and a haptic feedback subsystem 114.

The eye and/or head tracking subsystem 104 includes the appropriatehardware, software, firmware, sensors, and processing logic tocontinuously track the eye gaze direction or head position andorientation of the occupant 102 in real-time during operation of thevehicle. In certain embodiments, the tracking subsystem 104 includes orcooperates with eye tracking technology that determines the current andactual line-of-sight direction (vector) of the occupant 102. In layman'sterms, eye tracking technology can be utilized by the tracking subsystemto monitor where the occupant 102 is looking at any given moment. Thesystem 100 may utilize any suitably configured eye tracking subsystem104, whether currently available or developed in the future. Inpractice, an appropriate eye tracking subsystem 104 may use videotechnology, graphics processing, on-eye technology, head-worn displaysystem technology, NTE display system technology, or the like.

In some embodiments, the tracking subsystem 104 includes or cooperateswith head tracking or facial recognition technology that determines thecurrent and actual line-of-sight direction of the occupant 102 based onhead or face position, orientation, etc. The system 100 may utilize anysuitably configured head tracking subsystem 104, whether currentlyavailable or developed in the future. In practice, an appropriate headtracking subsystem 104 may use video technology, graphics processing,on-eye technology, head-worn display system technology, NTE displaysystem technology, or the like. Depending on the particular embodiment,the system 100 may leverage both a head tracking subsystem and an eyetracking subsystem if so desired.

The visual target monitor and/or tracking subsystem 106 includes theappropriate hardware, software, firmware, sensors, and processing logicto continuously receive, calculate, determine, or otherwise obtain dataassociated with a visual target of interest. As used here, a visualtarget may be something onboard the vehicle or something external to thevehicle. Moreover, a visual target may be stationary relative to thevehicle (e.g., a gauge or display element within the cockpit of anaircraft) or moving relative to the vehicle (e.g., a neighboring vehicleor a ground-based object such as a building, a mountain, or an airport).Thus, the subsystem 106 may be suitably configured to receive,calculate, or process real-time positioning data that indicates thelocation of the visual target using a known reference coordinate system.For example, the subsystem 106 may report the position of the visualtarget using GPS coordinates.

For known objects located in fixed positions onboard the vehicle (suchas instruments, display elements, control devices, indicator lights, andbuttons), the visual target monitor and/or tracking subsystem 106 maystore or access a database of position data that indicates the locationof various items relative to a known head or eye position of theoccupant 102. The known position data of such fixed objects can be usedin the manner described in more detail below to calculate desiredline-of-sight vectors from the occupant 102 to the fixed objects.

For moving objects located outside the vehicle, the visual targetmonitor and/or tracking subsystem 106 may cooperate with one or moreother onboard subsystems or external subsystems to receive the positiondata of visual targets in the exterior viewspace. The received orcalculated position data of moving objects can be used in the mannerdescribed in more detail below to calculate desired line-of-sightvectors from the occupant 102 to the moving targets of interest. Forexample, the subsystem 106 may cooperate and communicate with an airtraffic control datalink system to receive and process informationreceived from ground based air traffic control stations and equipment.In this context, the received information may include position datacorresponding to neighboring aircraft in the vicinity of the hostaircraft. As another example, the subsystem 106 may cooperate andcommunicate with a terrain avoidance and warning system (TAWS) thatsupplies data representative of the location of terrain that may be apotential threat to the aircraft. As yet another example, the subsystem106 may cooperate and communicate with a traffic and collision avoidancesystem (TCAS) that provides data associated with other aircraft in thevicinity, which may include speed, direction, altitude, and other statusdata. Moreover, the subsystem 106 may cooperate and communicate with anAutomatic Dependent Surveillance Broadcast (ADS-B) system that receivespositioning and status information directly from neighboring aircraft.

For stationary objects located outside the vehicle, the visual targetmonitor and/or tracking subsystem 106 may cooperate with one or moreother onboard subsystems or external subsystems to receive navigationdata, mapping data, GPS data, and/or other information that correspondsto the location of the stationary objects. For example, the subsystem106 may include or access a database populated with known GPScoordinates of known waypoints, points of interest, airports, landmarks,cities, bodies of water, and the like. The known position data ofstationary objects outside the vehicle can be used in the mannerdescribed in more detail below to calculate desired line-of-sightvectors from the occupant 102 to the stationary objects.

The line-of-sight vector calculation and analysis module 108 may beimplemented as a logical processing module that carries out thenecessary processing tasks in response to the execution of suitablywritten computer-executable software instructions. In this regard, themodule 108 may be realized using software, firmware, processing logic,or any combination thereof. The module 108 cooperates with the eyeand/or head tracking subsystem 104 and the visual target monitor and/ortracking subsystem 106 as needed to obtain the information used tocalculate and analyze the line-of-sight vectors in real-time. Theinformation received or accessed by the module 108 may include sensordata collected by the eye and/or head tracking subsystem 104, locationor positioning data provided by the visual target monitor and/ortracking subsystem 106, known location or positioning data (which may bepreloaded into the system 100) corresponding to stationary objectsinside the host vehicle, and/or other types of data that may be neededto support the visual assistance techniques and methodologies presentedhere.

The line-of-sight vector calculation and analysis module 108 is suitablyconfigured to determine whether or not the observer is gazing at theintended or desired visual target, based on the current line-of-sightvector that represents the actual gaze direction of the observer, andfurther based on the current line-of-sight vector that represents thedesired gaze direction from the observer to the visual target ofinterest. If the system 100 determines that the observer is likely to belooking at the visual target of interest, then little to no action needbe taken. In contrast, if the system 100 determines that the observer isgazing in the wrong direction (i.e., is not looking at the visualtarget), then at least one type of notification may be generated.

FIG. 2 is a diagram that depicts an in-flight scenario where the pilot202 of an aircraft needs assistance to direct his gaze toward a visualtarget of interest. FIG. 2 is a simplified rendition of the inside ofthe aircraft cockpit 204, which includes a number of display elements,instrumentation, controls, etc. FIG. 2 corresponds to an in-flightscenario during which the pilot 202 is currently looking straight out ofthe window 206 of the aircraft. Accordingly, the current and actualline-of-sight vector 208 for the pilot 202 is generally directed towardthe airspace immediately in front of the pilot 202.

FIG. 2 also depicts a neighboring aircraft 210 in flight within thevisible airspace near the host aircraft. If the neighboring aircraft 210is considered to be the visual target of interest, then the current viewstate of the pilot 202 should be corrected such that his actual gazedirection is toward the neighboring aircraft 210 rather than directly infront of him. In this regard, the difference between the actualline-of-sight vector 208 of the pilot 202 and the desired line-of sightvector 212 from the pilot 202 to the neighboring aircraft 210 is largeenough to cause the system 100 to generate a visual guidancenotification for the pilot 202. FIG. 2 also illustrates how an iteminside the cockpit, such as a control switch 214, may be the designatedvisual target of interest. Here, the difference between the actual theactual line-of-sight vector 208 of the pilot 202 and the desired line-ofsight vector 216 from the pilot 202 to the control switch 214 may alsobe large enough to cause the system 100 to generate a visual guidancenotification for the pilot 202.

Referring again to FIG. 1, the system 100 may generate any type ofnotification, as appropriate to the particular embodiment, intendedapplication, current operating environment and status, and the like. Thenotification provides guidance or assistance to redirect the currentgaze direction of the occupant toward the visual target of interest. Thesystem 100 may accomplish this goal in any number of different ways. Forexample, the system 100 may include or cooperate with the displaysubsystem 110 to generate and render visual notifications including,without limitation: alphanumeric messages; icons; directional indicatorssuch as arrows or chevrons; images; videos; animations; warning or alertindicators such as flashing lights or graphical representations offlashing lights; or the like. Additionally or alternatively, the system100 may include or cooperate with the audio subsystem 112 to generateand emit audible notifications including, without limitation: recordedor synthesized voice instructions; audio tones emitted from a locationor locations that are intended to direct the observer's attention towardthe sound source; warning or alert tones; or the like. Additionally oralternatively, the system 100 may include or cooperate with the hapticfeedback subsystem 114 to produce and initiate haptic feedback that isintended for the observer. The haptic feedback may include, withoutlimitation: vibrations or “knocks” transmitted to a yoke, joystick,steering wheel, shift knob, or other handheld element typically foundwithin the cabin or cockpit of the vehicle; vibrations or “knocks”transmitted to headgear, eyewear, or other equipment worn by theobserver; vibrations or “knocks” transmitted to the occupant's seat;vibrations or “knocks” transmitted to the floor; the actuation oractivation of dynamic or mechanical elements that can be felt by theobserver; etc. In practice, the haptic feedback is intelligentlygenerated in a way that conveys or otherwise indicates a viewspacelocation of the visual target of interest. For example, the right templeof the occupant's eyewear and/or the right side of the occupant's seatmay vibrate to indicate that the occupant should look to the right.

As mentioned above with reference to FIG. 1, the system 100 ispreferably deployed onboard the host vehicle. In an aircraftimplementation, the system 100 may cooperate with or be implemented as aflight deck computing system. In this regard, FIG. 3 is a simplifiedrepresentation of an exemplary embodiment of an onboard aircraftcomputing system 300 that provides visual search assistance to anoccupant 301 of the aircraft. In exemplary embodiments, the system 300is located onboard the host aircraft, i.e., the various components andelements of the system 300 reside within the host aircraft, are carriedby the host aircraft, or are attached to the host aircraft. Theillustrated embodiment of the system 300 includes, without limitation: aprocessor architecture 302 having at least one processor device; anappropriate amount of memory 304; at least one display element 306; anotification and alerting subsystem 308; a user interface 310; a datacommunication module 312; a datalink subsystem 314; at least one sourceof flight status data 316; and a tracking subsystem 320. These elementsof the system 300 may be coupled together by a suitable interconnectionarchitecture 322 that accommodates data communication, the transmissionof control or command signals, and/or the delivery of operating powerwithin the system 300. It should be understood that FIG. 3 is asimplified representation of the computing system 300 that will be usedfor purposes of explanation and ease of description, and that FIG. 3 isnot intended to limit the application or scope of the subject matter inany way. In practice, the system 300 and the host aircraft will includeother devices and components for providing conventional and traditionalfunctions and features. Furthermore, although FIG. 3 depicts the system300 as a single unit, the individual elements and components of thesystem 300 could be implemented in a distributed manner using any numberof physically distinct pieces of hardware or equipment.

The processor architecture 302 may be implemented or realized with anynumber of discrete processor devices, content addressable memory,digital signal processors, application specific integrated circuits,field programmable gate arrays, any suitable programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination designed to perform the functions described here. Aprocessor device may be realized as a microprocessor, a controller, amicrocontroller, or a state machine. Moreover, a processor device may beimplemented as a combination of computing devices, e.g., a combinationof a digital signal processor and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with adigital signal processor core, or any other such configuration. Theprocessor architecture is capable of executing computer-executableinstructions that cause the system 300 to perform the varioustechniques, operations, and functions related to the provision of visualguidance assistance.

The memory 304 may be realized as RAM memory, flash memory, EPROMmemory, EEPROM memory, registers, a hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. In thisregard, the memory 304 can be coupled to the processor architecture 302such that the processor architecture 302 can read information from, andwrite information to, the memory 304. In the alternative, the memory 304may be integral to the processor architecture 302. As an example, theprocessor architecture 302 and the memory 304 may reside in an ASIC. Inpractice, a functional or logical module/component of the computingsystem 300 might be realized using program code that is maintained inthe memory 304. For example, the notification and alerting subsystem308, the data communication module 312, the datalink subsystem 314,and/or the tracking subsystem 320 may have associated software programcomponents that are stored in the memory 304. Moreover, the memory 304can be used to store data utilized to support the operation of thesystem 300, as will become apparent from the following description.

In an exemplary embodiment, display elements 306 may be driven by asuitably configured graphics processor or graphics processing system(not shown). In this manner, the display elements 306 can be used todisplay, render, or otherwise convey one or more graphicalrepresentations, synthetic displays, graphical icons, visual symbology,textual messages, or images associated with operation of the hostaircraft and, in particular, the presentation of visual guidancenotifications as described in greater detail below.

The computing system 300 also includes or cooperates with thenotification and alerting subsystem 308. For this embodiment,notification and alerting subsystem 308 may be responsible for thegeneration and presentation of audio, visual, haptic, and/or other typesof feedback intended to help focus the occupant of the vehicle on adesired visual target. The notification and alerting subsystem 308 mayalso be responsible for the generation and presentation of other alerts,alarms, messages, and notifications associated with conventional androutine operations of the host vehicle.

The illustrated embodiment of the computing system 300 includes a userinterface 310, which is suitably configured to receive input from a user(e.g., a pilot, an operator, a crew member, or any occupant of thevehicle) and, in response to the user input, supply appropriate commandsignals to the system 300. The user interface 310 may be any one, or anycombination, of various known user interface devices or technologies,including, but not limited to: a cursor control device such as a mouse,a trackball, or joystick; a keyboard; buttons; switches; or knobs.Moreover, the user interface 310 may cooperate with one or more of thedisplay elements 306 to provide a graphical user interface.

In certain exemplary embodiments, the data communication module 312 issuitably configured to support data communication between the hostaircraft and one or more remote systems. More specifically, the datacommunication module 312 is used to receive current status data 324 ofother aircraft that are near the host aircraft. In particularembodiments, the data communication module 312 is implemented as anaircraft-to-aircraft data communication module that receives flightstatus data and GPS data from an aircraft other than the host aircraft.For example, the data communication module 312 may be configured forcompatibility with Automatic Dependent Surveillance-Broadcast (ADS-B)technology, with Traffic and Collision Avoidance System (TCAS)technology, and/or with similar technologies.

The status data 324 may include, without limitation: airspeed data;groundspeed data; altitude data; attitude data, including pitch data androll data; yaw data; geographic position data, such as GPS data;time/date information; heading information; weather information; flightpath data; track data; radar altitude data; geometric altitude data;wind speed data; wind direction data; etc. The computing system 300 issuitably designed to process the received status data 324 as needed andappropriate to determine a real-time eye gaze vector from the observerto the desired visual target of interest, e.g., a neighboring aircraft.

The datalink subsystem 314 enables the host aircraft to communicate withAir Traffic Control (ATC). In this regard, the datalink subsystem 314may be used to provide ATC data to the host aircraft and/or to sendinformation from the host aircraft to ATC, preferably in compliance withknown standards and specifications. In addition to conventional andtraditional uses, the system 300 may use the datalink subsystem 314 toobtain status and positioning data of neighboring aircraft (via ATCpersonnel or equipment) and/or positioning data of other visual targetsin the vicinity of the host aircraft.

In operation, the computing system 300 is also configured to process thecurrent flight status data for the host aircraft. In this regard, thesources of flight status data 316 generate, measure, and/or providedifferent types of data related to the operational status of the hostaircraft, the environment in which the host aircraft is operating,flight parameters, and the like. In practice, the sources of flightstatus data 316 may be realized using line replaceable units (LRUs),transducers, accelerometers, instruments, sensors, and other well-knowndevices. The data provided by the sources of flight status data 316 mayinclude, without limitation: airspeed data; groundspeed data; altitudedata; attitude data, including pitch data and roll data; yaw data;geographic position data, such as GPS data; time/date information;heading information; weather information; flight path data; track data;radar altitude data; geometric altitude data; wind speed data; winddirection data; etc. The computing system 300 is suitably designed toprocess data obtained from the sources of flight status data 316 in themanner described in more detail herein. In addition to conventional andtypical processing, the system 300 may leverage the flight status dataof the host aircraft to calculate line-of-sight vectors from theobserver to visual targets.

The tracking subsystem 320 corresponds to that described above withreference to FIG. 1. More specifically, the tracking subsystem 320represents the hardware, software, firmware, and/or processing logicthat is responsible for monitoring and tracking (in real-time) theobserver's eye gaze, head position or orientation, etc. Thefunctionality of the tracking subsystem 320 was described above withreference to the eye and/or head tracking subsystem 104.

FIG. 4 is a flow chart that illustrates an exemplary embodiment of avisual search assistance process 400, which may be performed by thesystems 100, 300 described above. The various tasks performed inconnection with the process 400 may be performed by software, hardware,firmware, or any combination thereof. For illustrative purposes, thefollowing description of the process 400 may refer to elements mentionedabove in connection with FIGS. 1-3. In practice, portions of the process400 be performed by different elements of the described system, e.g., atracking subsystem, a display subsystem, a data communication module, orthe like. It should be appreciated that the process 400 may include anynumber of additional or alternative tasks, the tasks shown in FIG. 4need not be performed in the illustrated order, and the process 400 maybe incorporated into a more comprehensive procedure or process havingadditional functionality not described in detail herein. Moreover, oneor more of the tasks shown in FIG. 4 could be omitted from an embodimentof the process 400 as long as the intended overall functionality remainsintact.

The process 400 may be initiated automatically based on the detection ofcertain operating conditions, alert conditions, or the like.Alternatively (or additionally), the process 400 may be initiated by auser of the system, either manually or by a voice command. For example,the process 400 might be triggered by a voice response system thatdetects a pilot query such as “Where is airport YYZ?” or “Where is thecircuit panel breaker?” or “Please locate the aircraft N123AB?” Suchtriggering mechanisms are desirable in situations where visual targetsoutside the host aircraft are not visible or are obscured from view.

The process 400 is preferably performed in an ongoing mannercontinuously in accordance with the desired sampling rate of the hostsystem. For example, an iteration of the process 400 could be performedonce every 20 milliseconds (or at any desired rate). Executing theprocess 400 in this dynamic manner is desirable to update the visualassistance system in real-time or substantially real-time for practicalpurposes. The illustrated embodiment of the process 400 begins byidentifying or otherwise obtaining the visual target of interest (task402). For example, task 402 may identify a neighboring aircraft as thevisual target, based on proximity to the host aircraft. As anotherexample, an onboard alerting or diagnostic subsystem may cooperate withthe process 400 as needed to identify a visual target inside the cockpitthat might require visual attention (e.g., an alert message, anindicator light, a display screen, or the like). The particular visualtarget that is identified at task 402, and the manner in which theprocess 400 identifies the visual target, may vary from one embodimentto another, from one operating environment to another, and from vehicleto another.

The process 400 also receives, calculates, or otherwise obtains thecurrent position/location of the visual target of interest (task 404).In accordance with certain embodiments, task 404 leverages GPStechnology to obtain position data corresponding to the visual target.The raw GPS coordinates may be broadcast from the visual target,provided by a terrestrial station (e.g., an ATC station), or the like.The position of the visual target is utilized to dynamically determinethe desired line-of-sight vector from an occupant (observer) of thevehicle to the visual target (task 406). In practice, task 406 mayprocess or determine a reference endpoint for the desired line-of-sightvector, where the endpoint corresponds to the position of the observer'seyes, face, or head. The reference endpoint may be an estimated orassumed position that represents a typical or average eye position of apilot or driver of the vehicle, or it may be a calculated or calibratedposition that is determined for each individual, perhaps at thebeginning of a flight, drive, or mission. In certain embodiments, aprecise reference endpoint may be obtained using the onboard eye gazetracking subsystem, using sensors associated with an NTE display system,or the like. In such embodiments, the reference endpoint can becontinuously tracked in real-time (or substantially real time as needed)by the tracking subsystem.

If the visual target is a stationary object that resides within thevehicle, then the desired line-of-sight vector can be dynamicallydetermined based on a known position or location of the visual targetrelative to the occupant, relative to a reference location such as thepilot's or driver's seat, relative to a calibrated head or eye positionof the occupant, or the like. In certain preferred embodiments, thereference location corresponds to the current eye position as determinedin real-time (or substantially real time) by the tracking subsystem. Ifthe visual target is a stationary ground-based target that residesoutside the vehicle, then the desired line-of-sight vector can bedynamically determined based on a known geographical position of thevisual target, such as its GPS coordinates. If the visual target is amoving object that resides outside the vehicle, then the desiredline-of-sight vector can be dynamically determined based on real-timeposition data of the visual target, which may be received at task 404.

The process 400 may continue by dynamically determining the actualline-of-sight vector for the occupant (observer) of the vehicle (task408). Task 408 calculates or obtains the current and real-time gazedirection of the occupant using the appropriate onboard tracking systemof the vehicle, as explained above. Accordingly, the actualline-of-sight vector may be determined based on a current eye gazedirection of the occupant, as detected by an onboard eye gaze trackingsystem. Alternatively or additionally, the actual line-of-sight vectormay be determined based on a current head position or orientation of theoccupant, as detected by an onboard head tracking system. As yet anotherexample, the actual line-of-sight vector may be determined based on acurrent eye gaze direction of the occupant, as detected by an onboardNTE display system (e.g., smart eyewear or a head-worn HUD system).Notably, the desired line-of-sight vector determined at task 406 istemporally associated or correlated with the actual line-of-sight vectordetermined at task 408. In other words, the timing of the two vectors isthe same.

The two obtained line-of-sight vectors are compared against each otherto obtain an appropriate difference measurement (task 410). In practice,the vectors may need to be rotated, translated, or otherwise processedto facilitate the comparison. For example, the vectors may need to berotated in accordance with a known reference frame to obtain anintelligible and relevant difference measurement. In layman's terms, thedifference measurement indicates the extent by which the observer's eyegaze is diverted from the intended and desired visual target. Thus, ifthe observer is looking at or near the visual target, then thedifference measurement will be relatively low. In contrast, if theobserver is looking far away from the visual target, then the differencemeasurement will be relatively high. Depending on the embodiment, thedifference measurement may be a scalar quantity or a vector quantity, orit could be expressed in any suitable manner.

The process 400 may continue by checking whether the differencemeasurement exceeds a threshold value (query task 412). The threshold isselected to reduce false alerts and nuisance messages, while stillpreserving the desired functionality. If the difference measurement doesnot exceed the threshold value (the “No” branch of query task 412), thenthe process 400 may exit or return to task 402 to initiate the nextprocessing iteration. If, however, the difference measurement exceedsthe threshold value (the “Yes” branch of query task 412), then theprocess 400 continues by considering whether a notification is needed.

Although not always required, the embodiment described here checkswhether the occupant's attention has been diverted for a sufficientlylong period of time. For example, if a pilot's eyes are darting aroundthe cockpit for a brief period of time before settling on the desiredvisual target, then it may not be necessary to generate a notificationto guide the pilot to the visual target. On the other hand, if the pilothas been looking away from the visual target for more than severalseconds, then it may be appropriate to generate an alert or an audiblemessage in an attempt to redirect the pilot's gaze. Accordingly, theprocess 400 may check whether a current value of a running timer exceedsa time threshold (query task 414). In this regard, the timer may beinitialized as soon as the difference measurement exceeds the thresholdvalue (see query task 412). The time threshold is selected to reducefalse alerts and nuisance messages, while still preserving the desiredfunctionality.

If the current timer value is less than or equal to the time threshold(the “No” branch of query task 414), then the process 400 may return totask 402 to initiate the next processing iteration while still trackingthe same visual target. If, however, the timer value exceeds the timethreshold (the “Yes” branch of query task 414), then the process 400continues by generating an appropriately formatted notification,message, display, feedback, or annunciation (task 416). Regardless ofthe format and chosen delivery method, the notification providesguidance to redirect the gaze direction of the occupant toward thevisual target of interest. As mentioned previously with reference toFIG. 1, different types of notifications may be available to the process400. For instance, the notification produced at task 416 may be anaudible instruction that is emitted to inform the occupant of thevehicle of a viewspace location of the visual target. In this regard,the onboard audio system may be used to announce a message such as:“LOOK TO YOUR LEFT” or “AIRCRAFT APPROACHING FROM ABOVE” or “THE ALERTIS COMING FROM THE MAIN CONTROL PANEL—LOOK FURTHER LEFT AND SLIGHTLYDOWN”. As another example, the notification may be generated in the formof a directional indicator, arrow, or icon that is graphically displayedfor the occupant and is formatted or otherwise designed to point towardthe viewspace location of the visual target. In practice, a directionalindicator could be displayed on a primary display element, on a HUDsystem, on a NTE display system, or the like. Alternatively oradditionally, the notification may take the form of a textual messagethat describes, identifies, or indicates the viewspace location of thevisual target. In this regard, the audio messages provided as examplesabove could be displayed as alphanumeric messages on one or more displayelements or systems. As yet another example, the notification could takethe form of haptic feedback intended for the observer, wherein thehaptic feedback provides some type of physical stimulation that isintended to indicate the viewspace location of the visual target ofinterest.

As mentioned above, the process 400 can be repeated in an ongoing mannerto continuously track the eye gaze direction of the observer,continuously track the position of the visual target (if necessary), andcontinuously update the line-of-sight vectors in real-time orapproximately real-time. Thus, task 416 will lead back to task 402 if itis time to perform another iteration of the process 400. Frequentupdating of the line-of-sight vectors is desirable to keep the occupantwell informed during operation of the vehicle.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or embodiments described herein are not intended tolimit the scope, applicability, or configuration of the claimed subjectmatter in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the described embodiment or embodiments. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the scope defined by theclaims, which includes known equivalents and foreseeable equivalents atthe time of filing this patent application.

What is claimed is:
 1. A visual search assistance method for an occupantof a vehicle, the method comprising: dynamically determining a desiredline-of-sight vector from an occupant of the vehicle to a visual target;dynamically determining, with an onboard tracking system of the vehicle,an actual line-of-sight vector for the occupant of the vehicle, thedetermined actual line-of-sight vector being temporally associated withthe determined desired line-of-sight vector; comparing the determinedactual line-of-sight vector against the determined desired line-of-sightvector to obtain a difference measurement; and generating a notificationwhen the obtained difference measurement exceeds a threshold value, thenotification providing guidance to redirect a gaze direction of theoccupant toward the visual target.
 2. The method of claim 1, wherein:the visual target resides within the vehicle; and the desiredline-of-sight vector is dynamically determined based on a known positionof the visual target relative to the occupant.
 3. The method of claim 1,wherein: the visual target is a stationary ground-based target thatresides outside the vehicle; and the desired line-of-sight vector isdynamically determined based on a known geographical position of thevisual target.
 4. The method of claim 1, wherein: the visual target is amoving object that resides outside the vehicle; the method furthercomprises receiving, with an onboard system of the vehicle, real-timeposition data of the visual target; and the desired line-of-sight vectoris dynamically determined based on the received real-time position dataof the visual target.
 5. The method of claim 1, wherein: the onboardtracking system comprises an eye gaze tracking system; and the actualline-of-sight vector is dynamically determined based on a current eyegaze direction of the occupant, as detected by the eye gaze trackingsystem.
 6. The method of claim 1, wherein: the onboard tracking systemcomprises a head tracking system; and the actual line-of-sight vector isdynamically determined based on a current head position of the occupant,as detected by the head tracking system.
 7. The method of claim 1,wherein: the onboard tracking system comprises a near-to-eye displaysystem; and the actual line-of-sight vector is dynamically determinedbased on a current eye gaze direction of the occupant, as detected bythe near-to-eye display system.
 8. The method of claim 1, whereingenerating the notification comprises: emitting an audible instructionto inform the occupant of a viewspace location of the visual target. 9.The method of claim 1, wherein generating the notification comprises:displaying a directional indicator that points toward a viewspacelocation of the visual target.
 10. The method of claim 1, whereingenerating the notification comprises: displaying a textual message thatdescribes, identifies, or indicates a viewspace location of the visualtarget.
 11. The method of claim 1, wherein generating the notificationcomprises: producing haptic feedback intended for the occupant of thevehicle, the haptic feedback indicating a viewspace location of thevisual target.
 12. A tangible and non-transitory computer readablemedium having computer-executable instructions stored thereon andcapable of performing a method when executed by a processor, the methodcomprising: dynamically obtaining a desired line-of-sight vector from anoccupant of an aircraft to a visual target; dynamically obtaining, froman onboard tracking system of the aircraft, an actual line-of-sightvector for the occupant of the aircraft, the determined actualline-of-sight vector being temporally associated with the determineddesired line-of-sight vector; comparing the determined actualline-of-sight vector against the determined desired line-of-sight vectorto obtain a difference measurement; and generating a notification whenthe obtained difference measurement exceeds a threshold value, thenotification providing guidance to redirect a gaze direction of theoccupant toward the visual target.
 13. The computer readable medium ofclaim 12, wherein: the onboard tracking system comprises an eye gazetracking system; and the actual line-of-sight vector is dynamicallydetermined based on a current eye gaze direction of the occupant, asdetected by the eye gaze tracking system.
 14. The computer readablemedium of claim 12, wherein: the onboard tracking system comprises ahead tracking system; and the actual line-of-sight vector is dynamicallydetermined based on a current head position of the occupant, as detectedby the head tracking system.
 15. The computer readable medium of claim12, wherein: the onboard tracking system comprises a near-to-eye displaysystem; and the actual line-of-sight vector is dynamically determinedbased on a current eye gaze direction of the occupant, as detected bythe near-to-eye display system.
 16. The computer readable medium ofclaim 12, wherein generating the notification comprises: emitting anaudible instruction to inform the occupant of a viewspace location ofthe visual target.
 17. The computer readable medium of claim 12, whereingenerating the notification comprises: displaying a directionalindicator that points toward a viewspace location of the visual target.18. A computing system for a vehicle, the computing system comprising aprocessor architecture having at least one processor device, andcomprising memory having computer-executable instructions stored thereonthat, when executed by the processor architecture, cause the computingsystem to: operate an onboard tracking system of the vehicle todynamically obtain a current line-of-sight vector for an occupant of thevehicle; compare the obtained current line-of-sight vector against acurrent desired line-of-sight vector from the occupant of the vehicle toa visual target, the obtained current line-of-sight vector beingtemporally associated with the current desired line-of-sight vector; andprovide information when the obtained difference measurement exceeds athreshold value, the information intended for the occupant of thevehicle, and the information providing guidance to redirect a gazedirection of the occupant toward the visual target.
 19. The computingsystem of claim 18, wherein the information is provided in the form ofan audible instruction that directs the occupant to a viewspace locationof the visual target.
 20. The computing system of claim 18, wherein theinformation is provided in the form of a displayed directional indicatorthat points toward a viewspace location of the visual target.